System and method for adapting transmission rate of a multimedia streaming server using a &#34;virtual clock&#34;

ABSTRACT

A so-called “Virtual Clock” with varying frequency is provided for used by a multimedia streaming server to adapt its transmission rate dynamically to changing network conditions. The “Virtual Clock” system and method of the present invention compensates for a potential limitation of the Internet Real-time Transmission Protocol (RTP), that stamps every packet it delivers with a timestamp and expects the server using this timestamp to schedule the transmission of this particular packet accordingly. Consequently, the transmission rate is pre-determined by the encoded multimedia content when RPT is used. Using the “Virtual Clock” of the present invention, the streaming server has a mechanism to overcome this RTP limitation and can conduct transmission rate adaptation in a way that can balance the bandwidth requirement of the content with the bandwidth availability of the network.

The present invention relates to multimedia streaming over a networkMore particularly, the present invention relates to adapting thetransmission rate of streamed multimedia to changing network conditions.Most particularly, the present invention introduces the concept of a“Virtual Clock” as a mechanism for a streaming server to perform dynamictransmission rate adaptation in a way that balances the bandwidthrequirement of the content to be transmitted with the bandwidthavailable of the Internet.

The design and implementation of state-of-the-art streaming serversgenerally includes a constant-frequency clock that is essentially thesame as the computer clock of the computer hosting the serverapplication. Packet scheduling and transmission are carried outaccording to the constant rate of this clock. The transmission rate ispre-determined only by the encoded content. This is evidenced in theimplementation of Darwin Streaming Server, that was developed by Appleand its source code that is openly available to public, see, forexample, http://developer.apple.com/darwin/projects/streaming/.

Since the available bandwidth of packet switching networks istime-varying, it is necessary for a streaming application to be able toadjust its transmission rate according to network conditions. Currentlyavailable techniques for rate adaptation include layer switching andselective layer subscription.

In layer switching, the server maintains multiple copies of the samecontent but encoded with different qualities and therefore different bitrates. The server can dynamically switch between these copies (orlayers) to achieve rate adaptation.

In selective layer subscription, the server only store one copy of thecontent encoded by a scalable coding scheme such as Fine-GranularScalability (FGS) or other similar scheme. A scalable coding schemegenerates multiple accumulative layers that can be sequentially added upat the receiver side to get better and better decoded quality. In realtime, the server only transmits the sub-set of the layers that have beenexplicitly requested, i.e., subscribed to, by the receiver. When thereceiver changes its layer subscriptions according to perceived networkconditions, the rate adaptation is achieved. The latter scheme is widelyproposed for multicast and commonly referred to as receiver-driven layermulticasting.

The limitation of the above techniques is their adaptation granularity.Both schemes can only achieve coarse-grained rate adaptation. In otherwords, they can only adapt rates to a level that is not frequent enough.However, experiments have shown that network conditions can changedramatically over relatively small time scales due to dynamic backgroundtraffic or temporary degradation of a wireless link.

An adaptive playout technique has been proposed. In this technique, thereceiver dynamically changes video playout speed to avoid bufferstarvation or overflow in the event of network congestion. However, thistechnique is proposed only for the receiver side, and has no effect onpacket transmission over the network. In fact, a combination of thepresent invention with this proposed adaptive playout strategy mayresult in a more efficient and robust streaming technique.

Thus, a method that can achieve fine-grained rate adaptation instreaming applications is highly desirable. The present inventionprovides a “Virtual Clock” having variable frequency that can be used bya multimedia streaming server to dynamically adapt its transmission rateto changing network conditions. This “Virtual Clock” compensates for apotential limitation of the Internet Real-time Transmission Protocol(RTP), that stamps every packet it delivers with a timestamp and expectsthe server using this timestamp to schedule the transmission of thisparticular packet. Consequently, the transmission rate is pre-determinedby the encoded multimedia content when RTP is used. By providing a“Virtual Clock” according to the present invention, the multimediastreaming server has a mechanism to overcome this RTP limitation andperform transmission rate adaptation in a way that balances thebandwidth requirement of the content and the bandwidth availability ofthe network.

The “Virtual Clock” of the present invention addresses the issue offine-grained rate adaptation. A streaming server needs a clock toschedule the transmission of time-stamped RTP packets. If the clockmoves forward at a constant rate, then the transmission rate will bepre-determined by the RTP timestamps that are normally generated atcoding stage.

By contrast, a “Virtual Clock” according to the present invention,adopts a time-varying frequency. When such a clock is used by a serverto schedule transmissions, it provides a variable to be added to thetransmission rate that was pre-determined by the encoder. In this way,the transmission rate can be elastic in its response to changing networkconditions.

For example, assume the frequency for a real clock is 1 100, asillustrated in FIG. 1 a. As illustrated in FIGS. 1 b and 1 c,respectively, the “Virtual Clock” can take a frequency either larger 102or smaller 104 than 1. When the frequency of the “Virtual Clock” becomeslarger 104 than 1, it will move faster than the real clock. Then, evenif the timestamp sequences of a group of RTP packets remain unchanged,the intervals 101 between consecutive packets are shortened 103 by usingthe “Virtual Clock” to schedule them. The RTP packets appear at thenetwork interface more frequently than normal, leading to an increase inthe transmission rate over that pre-determined by the encoder. Bycontrast, when the “Virtual Clock” takes on a frequency smaller 104 than1, the intervals 101 between consecutive packets are lengthened 105 andthe packets appear at the network interface less frequently than normal,leading to a decrease in the transmission rate over that pre-determinedby the encoder. Whenever there is a change to the frequency of the“Virtual Clock”, there will be a correspond change in the transmissionrate. Therefore, the “Virtual Clock” according to the present invention,is an efficient system and method for streaming applications to adaptthe transmission rate of a sequence of time-stamped RTP packets tonetwork conditions.

Since the adjustment of the frequency of the “Virtual Clock” can becarried out over any time scale, particularly over small time scales,the “Virtual Clock” of the present invention can be used to achievefine-grained rate adaptation and is the most important characteristic of“Virtual Clock”. By combining the “Virtual Clock” with other methods,such as in the example presented above, a streaming server is able toadapt its transmission rate over both large and small time scales,achieving better responsiveness to dynamic network conditions. Improvedresponsiveness leads to better network resource utilization and bettervideo quality.

FIG. 1 a illustrates packet arrival time at the network interface for areal clock.

FIG. 1 b illustrates packet arrival time at the network interface for a“Virtual Clock” according to the present invention having a frequencygreater than that of the real clock illustrated in FIG. 1 a.

FIG. 1 c illustrates packet arrival time at the network interface for a“Virtual Clock” according to the present invention having a frequencyless than that of the real clock illustrated in FIG. 1 a.

Assume f(t) is the frequency of the “Virtual Clock”, R₀(t) is thepredetermined RTP packet rate, R_(L)(t) is the network bandwidth that isavailable to this streaming application, and the frequency of a realclock is 1. Also assume T is a time period in which both the real clockand the “Virtual Clock” advance the same distance in time space. That is$\begin{matrix}{T = {\int_{0}^{T}{{f(t)}\quad{\mathbb{d}t}}}} & (1)\end{matrix}$In a preferred embodiment, the frequency of the “Virtual Clock” isconfigured as follows $\begin{matrix}{{f(t)} = \left\{ {{\begin{matrix}{{R_{L}(t)}/{R_{0}(t)}} \\0\end{matrix}{when}\quad\begin{matrix}{t<=\tau} \\{t > \tau}\end{matrix}},{{{where}\quad\tau\quad{is}\quad{determined}\quad{by}\quad T} = {\int_{0}^{\tau}{{f(t)}\quad{\mathbb{d}t}}}}} \right.} & (2)\end{matrix}$The formula (1) prescribes a general principle about how to configurethe frequency of the “Virtual Clock” such that after every T time thetwo clocks re-synchronize, which is necessary for real-time streamingapplications.

R₀(t) is obtained from the encoded contents that are stored in theserver. R_(L)(t) 5 is measured by either the network interface driver atthe server, or some dedicated network components residing in the networkor at the receiver, and that calculates available bandwidth for thestreaming application.

For example, in the instance of in-home 200 streaming over wirelessillustrated in FIG. 2, due to radio frequency interference and channelfading, the wireless link capacity (such as R_(L)(t)) can change withtime. In a preferred embodiment illustrated in FIG. 2, a monitor isplaced into the wireless network driver 203 so that the driver measuresR_(L)(t) and sends the measurement back 205 to the streaming server 206allowing the transmission rate to be adapted to the wireless link statusin real time. In this way, unnecessary packet drops can be avoided andthe overall throughput can be improved.

In another preferred embodiment, in order to provide “Virtual Clock”service in parallel with real clock service to streaming applications bya host computer, a kernel function is implemented that has the formvoid getvirtualclockfrequency(double demandbandwidth,double*virtualfequency).

When invoked, this function interacts with the network card driver orlower layer protocols to return a virtual frequency to the server. Theserver then maps the real clock to the “Virtual Clock”.

As illustrated in FIG. 3, the “Virtual Clock” of the present inventioncan be implemented at the application layer 300, but its frequency iscontrolled by a lower layer, in a preferred embodiment this is the linklayer (or layer 2) 301. The link layer keeps monitoring the link status.If the available capacity is higher than a targeted capacity (a controlreference), then the link layer will send up a clock frequency f(t)) 302larger than 1, otherwise, smaller than 1.

The systems and methods of the present invention, as described above andshown in the drawings, provide for a ‘virtual ‘clock’ base on changingnetwork conditions. It will be apparent to those skilled in the art thatvarious modifications and variations can be made in the methods andsystems of the present invention without departing from the spirit orscope of the invention. Thus, it is intended that the present inventioninclude modifications and variations that are within the scope of theappended claims and their equivalents.

1. A communication network (207), comprising: a real clock (100) thatdetermines a pre-determined RTP packet transmission rate for a streamingapplication, R₀(t), based on encoded content; a real clock (102) (104)having a frequency f(t) that determines a dynamic transmission rate forthe streaming application; a streaming server (206) that transmits aplurality of RTP packets at the determined dynamic transmission rate forthe streaming application; and a network component (203) that calculatesavailable bandwidth R_(L)(t) (202) for the streaming application,wherein f(t) is dynamically adjusted based on R_(L)(t) (202) and R₀(t).2. The communication network (207) of claim 1, wherein the streamingserver (206) is a multimedia streaming server.
 3. The communicationnetwork (207) of claim 1, wherein the frequency f(t) of the real clock(102) (104) is configured as follows if the real clock (100) is assumedto have a frequency f(t)=1 and T is a time period in which both the realclock (100) and the real clock (102) (104) advance the same distance intime space, that is   T = ∫₀^(τ)f(t)  𝕕t then${f(t)} = \left\{ {{\begin{matrix}{{R_{L}(t)}/{R_{0}(t)}} \\0\end{matrix}{when}\quad\begin{matrix}{t<=\tau} \\{t > \tau}\end{matrix}{where}\tau\quad{is}\quad{determined}\quad{by}\quad T} = {\int_{0}^{\tau}{{f(t)}\quad{\mathbb{d}t}\quad{and}}}} \right.$R₀(t) is a pre-determined RTP packet rate based on content, wherein,after every T time the real clock (100) and the real clock (102) (104)re-synchronize.
 4. The communication network (207) of claim 3, whereinR_(L)(t) is measured by one of a network interface driver at thestreaming server (206), a set of one or more dedicated networkcomponents (203) residing in the network (207), and a set of one or morededicated components at a receiver.
 5. The communication network (207)of claim 4, wherein the network (207) is a wireless network and the setof one or more dedicated components at the receiver is a monitor placedinto the wireless network driver such that the driver measures R_(L)(t)(202) and sends the measured R_(L)(t) (202) to the streaming server(206).
 6. An apparatus for dynamically adjusting the transmission rateover a network (207) of a streaming server (206), comprising: a realclock (100) that determines a pre-determined RTP packet transmissionrate for a streaming application, R₀(t), based on encoded content; areal clock (102) (104) having a frequency f(t) that determines a dynamictransmission rate for the streaming application; and a network component(203) that calculates available bandwidth R_(L)(t) (202) for thestreaming application, wherein f(t) is dynamically adjusted based onR_(L)(t) (202) and f(t) (302).
 7. The apparatus of claim 6, wherein thestreaming server (206) is a multimedia streaming server.
 8. Theapparatus of claim 6, wherein the frequency f(t) of the real clock (102)(104) is configured as follows if the real clock (100) is assumed tohave a frequency f(t)=1 and T is a time period in which both the realclock (100) and the real clock (102) (104) advance the same distance intime space, that is   T = ∫₀^(τ)f(t)  𝕕t then${f(t)} = \left\{ {{\begin{matrix}{{R_{L}(t)}/{R_{0}(t)}} \\0\end{matrix}{when}\quad\begin{matrix}{t<=\tau} \\{t > \tau}\end{matrix}{where}\tau\quad{is}\quad{determined}\quad{by}\quad T} = {\int_{0}^{\tau}{{f(t)}\quad{\mathbb{d}t}\quad{and}}}} \right.$R₀(t) is a pre-determined RTP packet rate based on content, wherein,after every T time the real clock (100) and the real clock (102) (104)re-synchronize.
 9. The apparatus of claim 8, wherein R_(L)(t) ismeasured by one of a network interface driver at the streaming server(206), a set of one or more dedicated network components (203) residingin the network (207), and a set of one or more dedicated components at areceiver.
 10. The apparatus of claim 9, wherein the network (207) is awireless network (207) and the set of one or more dedicated componentsat the receiver is a monitor placed into the wireless network driversuch that the driver measures R_(L)(t) (202) and sends the measuredR_(L)(t) (202) to the streaming server (206).
 11. A real clock (102)(104) for enabling a streaming server (206) to perform dynamictransmission rate adaptation, comprising: a real clock (100) thatdetermines a pre-determined RTP packet transmission rate for a streamingapplication, R₀(t), based on encoded content; means for dynamicallysetting the frequency f(t) of the real clock (102) (104) that determinesthe rate of RTP packet transmission for the streaming application; and anetwork component (203) that calculates available bandwidth R_(L)(t)(202) for the streaming application, wherein f(t) (302) is dynamicallyadjusted based on R_(L)(t) (202) and R₀(t).
 12. The real clock (102)(104) of claim 11, wherein the streaming server (206) is a multimediastreaming server.
 13. The real clock (102) (104) of claim 11, whereinthe means for determining the frequency f(t) of the real clock (102)(104) is a module that configures the frequency of f(t) as follows ifthe real clock (100) is assumed to have a frequency f(t)=1 and T is atime period in which both the real clock (100) and the real clock (102)(104) advance the same distance in time space, that is  T = ∫₀^(τ)f(t)  𝕕t then ${f(t)} = \left\{ {{\begin{matrix}{{R_{L}(t)}/{R_{0}(t)}} \\0\end{matrix}{when}\quad\begin{matrix}{t<=\tau} \\{t > \tau}\end{matrix}{where}\tau\quad{is}\quad{determined}\quad{by}\quad T} = {\int_{0}^{\tau}{{f(t)}\quad{\mathbb{d}t}\quad{and}}}} \right.$R₀(t) is a pre-determined RTP packet rate based on content, wherein,after every T time the real clock (100) and the real clock (102) (104)re-synchronize.
 14. The real clock (102) (104) of claim 11, whereinR_(L)(t) is measured by one of a network interface driver at the server,a set of one or more dedicated network components (203) residing in thenetwork (207), and a set of one or more dedicated components at areceiver, and that calculates available bandwidth for the streamingapplication.
 15. The real clock (102) (104) of claim 11, wherein thenetwork (207) is a wireless network (207) and the set of one or morededicated components at the receiver is a monitor placed into thewireless network driver such that the driver measures R_(L)(t) (202) andsends the measured R_(L)(t) (202) to the streaming server (206).
 16. Anoperating system kernel function at an application layer (300) of aprotocol that implements the real clock (102) (104) of claim 13,wherein, the function interacts with a lower layer (301) of the protocolto return the virtual frequency f(t) (302).
 17. A method forimplementing a real clock (102) (104) for enabling a streaming server(206) to perform dynamic transmission rate adaptation for RTP packettransmission over a network (207), comprising the steps of: providing areal clock (100) that determines a pre-determined RTP packettransmission rate for a streaming application, R₀(t), based on encodedcontent; dynamically configuring the frequency f(t) of the real clock(102) (104) that determines the rate of RTP packet transmission for astreaming application; and monitoring the available bandwidth R_(L)(t)(202) for the streaming application, dynamically adjusting f(t) (302) isbased on R_(L)(t) (202) and R₀(t).
 18. The method of claim 17, whereinthe configuring step further comprises the steps of a. if the real clock(100) is assumed to have a frequency f(t)=1 and T is a time period inwhich both the real clock (100) and the real clock (102) (104) advancethe same distance in time space, that is   T = ∫₀^(τ)f(t)  𝕕tthen  calculating ${f(t)} = \left\{ {{\begin{matrix}{{R_{L}(t)}/{R_{0}(t)}} \\0\end{matrix}{when}\quad\begin{matrix}{t<=\tau} \\{t > \tau}\end{matrix}{where}\text{}\tau\quad{is}\quad{determined}\quad{by}\quad T} = {\int_{0}^{\tau}{{f(t)}\quad{\mathbb{d}t}\quad{and}}}} \right.$R₀(t) is a pre-determined RTP packet rate based on content, b. afterevery T time, re-synchronizing the real clock (100) and the real clock(102) (104).
 19. The method of claim 18, further comprising the step of:measuring R_(L)(t) by one of a network interface driver at the server, aset of one or more dedicated network components (203) residing in thenetwork (207), and a set of one or more dedicated components at areceiver, and that calculates available bandwidth for the streamingapplication.
 20. The method of claim 18, wherein the network (207) is awireless network (207); the set of one or more dedicated components atthe receiver is a monitor placed into the wireless network driver; themonitoring step further comprises the steps c. measuring R_(L)(t) (202)by the monitor R_(L)(t) (202); and d. sending the measured R_(L)(t)(202) to the streaming server (206).